27 research outputs found
Global convergence of the gradient method for functions definable in o-minimal structures
We consider the gradient method with variable step size for minimizing
functions that are definable in o-minimal structures on the real field and
differentiable with locally Lipschitz gradients. We prove that global
convergence holds if continuous gradient trajectories are bounded, with the
minimum gradient norm vanishing at the rate if the step sizes are
greater than a positive constant. If additionally the gradient is continuously
differentiable, all saddle points are strict, and the step sizes are constant,
then convergence to a local minimum holds almost surely over any bounded set of
initial points.Comment: 33 pages, 1 figur
Lyapunov stability of the subgradient method with constant step size
We consider the subgradient method with constant step size for minimizing
locally Lipschitz semi-algebraic functions. In order to analyze the behavior of
its iterates in the vicinity of a local minimum, we introduce a notion of
discrete Lyapunov stability and propose necessary and sufficient conditions for
stability.Comment: 11 pages, 2 figure
Certifying the absence of spurious local minima at infinity
When searching for global optima of nonconvex unconstrained optimization
problems, it is desirable that every local minimum be a global minimum. This
property of having no spurious local minima is true in various problems of
interest nowadays, including principal component analysis, matrix sensing, and
linear neural networks. However, since these problems are non-coercive, they
may yet have spurious local minima at infinity. The classical tools used to
analyze the optimization landscape, namely the gradient and the Hessian, are
incapable of detecting spurious local minima at infinity. In this paper, we
identify conditions that certify the absence of spurious local minima at
infinity, one of which is having bounded subgradient trajectories. We check
that they hold in several applications of interest.Comment: 31 pages, 4 figure
Nonsmooth rank-one matrix factorization landscape
We provide the first positive result on the nonsmooth optimization landscape
of robust principal component analysis, to the best of our knowledge. It is the
object of several conjectures and remains mostly uncharted territory. We
identify a necessary and sufficient condition for the absence of spurious local
minima in the rank-one case. Our proof exploits the subdifferential regularity
of the objective function in order to eliminate the existence quantifier from
the first-order optimality condition known as Fermat's rule.Comment: 23 pages, 5 figure
Sufficient conditions for instability of the subgradient method with constant step size
We provide sufficient conditions for instability of the subgradient method
with constant step size around a local minimum of a locally Lipschitz
semi-algebraic function. They are satisfied by several spurious local minima
arising in robust principal component analysis and neural networks.Comment: 14 pages, 3 figure
Global stability of first-order methods for coercive tame functions
We consider first-order methods with constant step size for minimizing
locally Lipschitz coercive functions that are tame in an o-minimal structure on
the real field. We prove that if the method is approximated by subgradient
trajectories, then the iterates eventually remain in a neighborhood of a
connected component of the set of critical points. Under suitable
method-dependent regularity assumptions, this result applies to the subgradient
method with momentum, the stochastic subgradient method with random reshuffling
and momentum, and the random-permutations cyclic coordinate descent method.Comment: 30 pages, 1 figur